As the Internet has grown from a few servers controlled by the government and a few educational institutions into a vast, heterogeneous network of servers and clients, the demands on servers have grown as well. Servers connected to the Internet must often deal with thousands of requests every second, as users from around the world seek to access data. In a typical server topology, web servers respond to basic user inquiries and forward particular requests to application servers; these servers host applications with which the users interact, and have access to database servers in which application data is stored. Web servers, application servers and database servers are often not running on the same physical computers. Consequently, application servers, which respond to forwarded users' inquiries, execute application code and exchange data with database servers, must implement increasingly powerful hardware to keep pace with the demands placed on them. Although rapid gains have been made in both networking technology and processor speed, recent networking improvements have vastly outstripped the computer system improvements necessary to take advantage of them.
Traditionally, in order to send information across a back-end network, application servers exchange data packaged according to various network transport protocols with the database servers, encoding and decoding them as necessary to extract the relevant information. According to the standard networking, Open System Interconnect (OSI) model, there are seven layers through which a transmission travels: application layer, presentation layer, session layer, transport layer, network layer, data link layer and physical layer. Using legacy network devices and drivers, software processes performed by the CPU implement all but the final two network layers (data link and physical), which are implemented on the networking hardware itself. As a result, in addition to managing applications and application requests, an application server must dedicate processing time to the relatively simple but time-consuming network functionality. Moreover, these OSI processes also waste the valuable system resources necessary to copy information between memory buffers. In these legacy systems, servers are not limited by the bandwidth of the underlying network connecting them, but rather are limited by their processors' ability to balance application usage with networking functions.
One solution to this problem is presented by system area network technology. A system area network (SAN) is defined as a high-performance, connection-oriented network that provides high-bandwidth and low-latency characteristics to its nodes, often servers. Importantly, in addition to the high-speed connections and the high-speed routing technology, SANs employ specially designed network hardware, or network interface cards (NICs) to take advantage of new transfer protocols. Most important among these protocols is remote direct memory access (RDMA), a protocol defining a method by which a compatible NIC can directly send data to and receive data from remote memory connected to the SAN through another compatible NIC. Thus, the RDMA protocol avoids wasting the processor cycles required to encode and decode transferred data by offloading these processes to the RDMA-compatible NIC. Since the NIC becomes responsible for the packaging, flow, error checking of the data and even the transfer of the data to an appropriate memory buffer, the server's CPU cycles are freed from these tasks to provide more application resources. In this way, network performance (measured by how many requests can be handled in a given period of time) can be markedly improved without requiring a corresponding improvement in CPU speed.
RDMA itself defines a complex set of protocols that a compatible NIC and computer system must adhere to. Prior to sending or receiving data, a server using an RDMA-compatible NIC must register memory buffers with the NIC. These registered buffers then become the memory locations that can be directly accessed from any RDMA-compatible NIC communicating with the local memory controller. This initial registration is relatively resource-intensive but prevents the overwriting of sensitive information. As the NIC continues to communicate using RDMA commands, the initially registered memory buffers may be de-registered and new buffers registered to send and receive data.
Exploiting RDMA functionality, a compatible NIC can send and receive data using three sets of commands: Read operations, Write operations, and Post Send/Post Receive operations. A computer hosting long-term information can generically be referred to as a server, while the computer accessing that information (writing or reading it) can be referred to as a client. When a client wishes to read information stored on a server, the respective RDMA-compatible NICs first communicate regarding what information should be read, and the server sends an identifier for the registered memory location containing that information to the client's NIC. The client's NIC then sends an RDMA Read request to the server's NIC identifying the location of the stored information. The server's NIC receives the request, processes it, directly accesses the stored information at the memory location using the memory controller and forwards the information to the client's NIC, which processes the information and stores the information in its own registered memory buffer. Upon completion, the server's NIC sends a message to the client's NIC that the Read operation was completed successfully. The Write operation is similar in operation. In order to write information onto a server, a client's NIC first requests information from the server's NIC regarding where information should be written. Upon receiving virtual address information, the client's NIC sends a Write RDMA command along with the data to the server's NIC, which directly writes the data to the address indicated by the transmission. Upon completion, the server's NIC sends a message to the client's NIC indicating that the Write operation was completed successfully.
Although these commands are exceptionally quick and avoid many of the drawbacks of legacy networks, there are security issues raised by disclosing the memory locations to which other computers can write directly. As a result, two related commands are defined, Post Send and Post Receive. Whereas in normal RDMA Read and Write operations, the NIC sending the command specifies both the originating and destination addresses, in Post Send and Post Receive mode, the sending computer's NIC (whether client or server) specifies the address from which the information will be sent, while the receiving computer's NIC (whether client or server) specifies the address at which the information will be received. In this more secure implementation, the machines originally communicate concerning the size and content of the message, but neither machine knows the details of the other's internal memory buffers.
The benefits of SANs, and associated RDMA protocols are clear. Using this technology, an application server with less computing power can provide information access to a greater number of users by offloading network protocol tasks to RDMA-compatible NIC hardware, thus freeing CPU cycles and obviating unnecessary data duplication. However, having developed this new networking technology, a difficulty remains in coordinating it with existing applications and protocols. If every application were rewritten to comply with SAN technology, this would not only represent a huge undertaking, but in addition, two sets of application would have to be employed, one ported for legacy hardware and one for SAN hardware. Moreover, if the SAN hardware ever failed, the newly written applications would be useless. On the other hand, if operating system-level protocols were simply redesigned to take advantage of this new SAN technology, the task of rewriting every application may be avoided, but, if the SAN hardware failed, the network communications would still fail if support for legacy protocols were removed. As used in this specification, legacy protocols refer to those networking protocols used with network interfaces before system area networks were implemented. Typically, these protocols, as described above, require a great deal of software processing before the network hardware implements the most basic data link and physical layer functionality.
Therefore, there is a need for an I/O protocol that can take advantage of the new SAN technology while retaining compatibility with older programs and older legacy network technology. Such a system would enable older programs to take advantage of the speed improvements inherent in RDMA, while retaining compatibility with older network interfaces, to protect against SAN failure and to provide access to older network devices.